Maunsell,  John 

1  October  1989  -  30  September  1992 


N00014.90.J-1070 

Final  Technical  Report 


Overview:  The  work  in  this  project  was  directed  at  trying  to  understand  the  brain  mechanisms  that 
subserve  higher  visual  functions.  The  approach  used  involved  examining  the  activity  of  neurons 
in  the  brains  of  nonhuman  primates  while  they  are  performing  visual  tasks  that  they  have  been 
trained  to  do.  One  of  our  primary  interests  is  how  populations  of  neurons  create  representations 
that  support  short  term  visual  memories.  To  examine  this  question,  we  trained  monkeys  to 
perform  matching  tasks  in  which  they  are  shown  visual  stimuli  that  they  must  remember  for  short 
periods.  We  have  found  that  many  neurons  in  visual  cerebral  cortex  are  selectively  activated 
during  periods  when  the  animal  is  keeping  a  particular  stimulus  in  mind.  The  activity  of  these 
neurons  is  likely  to  provide  the  basis  for  short  term  memories,  and  understanding  the  origin  of  this 
activity  and  its  interactions  with  sensory  signals  will  be  an  important  step  in  understanding  how 
the  brain  implements  higher  functions  such  as  visual  search. 

General  Background:  Our  earlier  work  showed  that  memory-related  signals  can  be  found  even  at 
relatively  early  stages  of  processing  in  visual  cortex.  The  visual  cortex  in  all  primates,  including 
man,  is  organized  in  a  hierarchical  fashion.  Signals  coming  from  the  retinas  in  the  eyes  arrive  at 
the  primary  visual  cortex  (striate  cortex,  or  V 1),  which  performs  an  initial  analysis  and  then  relays 
the  information  to  other  areas  in  visual  cortex.  These  other  areas  perform  further  analysis  and 
relay  the  results  to  still  higher  areas.  The  process  is  repeated  many  times,  culminating  in 
representations  in  the  highest  visual  areas  in  the  parietal  and  temporal  lobes.  In  the  rhesus  monkey 
there  are  known  to  be  at  least  30  discrete  areas  in  visual  cortex,  each  representing  a  different  type 
of  information  about  the  visual  world. 


Changes  in  neuronal  activity  related  to  memory,  attention  or  motivation  have  been  described  by 
many  laboratories  in  the  highest  levels  of  visual  cortex,  but  whether  they  influence  information 
processing  in  earlier  stages  had  not  been  extensively  examined.  For  this  reason,  we  examined 
memory  related  activity  in  area  V4,  which  is  at  an  intermediate  level  of  cortical  processing,  only 
two  steps  removed  from  the  primary  visual  cortex.  We  drained  rhesus  monkeys  to  perform  a  task 
in  which  they  had  to  remember  the  orientation  of  a  briefly  presented  grating  pattern,  and  we 
compared  the  activity  of  individual  neurons  in  V4  while  the  animals  remembered  different 
orientations.  We  found  that  as  many  as  half  of  the  neurons  in  this  area  had  different  levels  of 
activity  depending  on  what  the  animal  was  remembering,  with  each  neuron  most  active  while  the 
animal  remembered  a  particular  orientation.  Collectively,  these  neurons  provided  an  unambiguous 
signal  about  the  orientation  that  the  animal  was  remembering.  It  is  likely  that  they  contribute  in  an 
important  way  to  the  ability  to  remember  different  orientations. 

Memory  related  signals  in  visual  cortex  provide  a  particularly  valuable  avenue  for  exploring  the 
neural  mechanisms  of  a  higher  brain  function.  Memory  related  neuronal  activity  can  be  examined 
during  periods  when  no  stimulus  is  present,  and  when  no  motor  acts  are  being  performed.  This 
eliminates  potential  confounding  factors  and  greatly  simplifies  interpretation  of  results. 


Progress:  During  the  funding  period  been  working  to  better  characterize  the  distribution  of  these 
signals  in  cerebrid  cortex,  with  the  long  term  goal  of  understanding  which  brain  structures  they 
depend  on  and  how  they  interact  with  sensory  information  in  generating  behaviors.  We  have 
focused  the  question  of  how  memory  related  signals  are  distributed  between  the  two  streams  of 
processing  that  exist  in  visual  cortex.  As  information  is  transmitted  to  higher  levels  of  cortical 
processing,  it  is  divided  into  two  streams  of  processing  that  subserve  different  visual  functions. 
Each  stream  is  made  up  of  many  cortical  areas.  One  leads  to  parietal  cortex,  and  is  important  for 
visual  assessment  of  spatial  relationships  and  movements.  The  other  leads  to  the  temporal  lobe 
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and  is  important  for  recognition  of  pattern  and  shape.  These  streams  have  also  been  called  the 
“motion  pathway”  and  the  “color  and  form  pathway.” 

Area  V4,  in  which  we  found  activity  related  to  short  term  memories,  lies  in  the  teniporal 
(color/form)  pathway.  We  wanted  to  see  if  similar  memory  related  activity  could  be  found  in  the 
parietal  pathway,  which  subserves  spatial  and  motion  analysis.  To  do  this  we  recorded  from 
neurons  in  the  middle  temporal  visuid  area  (MT),  which  is  an  important  element  in  the  parietal 
pathway.  We  selected  MT  because  it  lies  at  the  same  hierarchical  level  of  processing  as  V4,  arid 
because  the  stimulus  preferences  of  its  neurons  have  been  extensively  characterized.  Neurons  in 
MT  are  very  sensitive  to  the  direction  of  stimulus  motion.  They  respond  strongly  when  a  stimulus 
moves  through  their  receptive  field  in  the  preferred  direction,  and  not  at  all  if  the  same  stimulus 
moves  in  the  opposite  direction.  Because  MT  neurons  are  sensitive  to  direction,  we  used  a 
matching  task  in  which  the  animal  was  presented  with  patches  of  randomly-spaced  dots,  which 
moved  coherently  across  the  screen.  The  animals  had  to  remember  the  direction  of  a  briefly 
presented  random-dot  pattern,  and  we  examined  the  activity  of  neurons  in  MT  while  the  animals 
were  remembering  different  directions. 

We  found  few  memory  related  effects  in  MT.  We  therefore  went  on  to  examined  later  stages  of 
processing  in  the  parietal  pathway:  the  medial  superior  temporal  area  (MST),  which  receives 
output  from  MT;  and  area  7a,  which  is  one  of  the  highest  levels  of  processing  in  the  parietal 
pathway.  Although  these  areas  contained  neurons  with  some  memory-related  activity,  the  effects 
were  fewer  and  weaker  than  those  we  had  seen  in  V4  during  orientation  matching. 

This  absence  of  pronounced  effects  in  the  parietal  pathway  led  us  to  examine  responses  in  y4 
using  the  direction  matching  task.  Although  many  neurons  in  V4  have  a  weak  direction 
preference,  very  few  are  strongly  direction  selective,  and  we  therefore  did  not  expect  to  find  many 
V4  neurons  with  memory  related  effects  during  direction  matching.  To  our  surprise,  many  V4 
neurons  showed  memory  related  changes  in  activity  when  the  animal  was  remembering  different 
directions  of  motion.  TTie  effects  were  about  as  prevalent  as  those  seen  during  the  orientation 
matching  task.  Even  neurons  that  responded  equally  well  to  any  direction  of  stimidus  motion  were 
sometimes  much  more  or  less  active  while  the  animal  was  remembered  particular  directions  of 
motion.  These  results  were  reported  in  a  recent  article  in  the  Journal  of  Neuroscience,  which  is 
included  in  the  appendix. 

These  results  suggest  that  area  V4,  and  presumably  other  areas  in  the  temporal  pathway,  are 
important  in  tasla  that  involve  short  term  memory,  regardless  of  what  visual  stimulus  is  being 
remembered.  Although  the  parietal  pathway  is  generdly  associated  with  motion  processing,  it 
now  appears  that  it  is  not  involved  in  all  classes  of  motion  analysis.  We  believe  that  the  parietal 
pathway  may  be  specifically  adapted  to  serving  the  needs  of  visual  guidance  and  navigation,  and 
may  not  be  involv^  in  evaluating  other  types  of  movements  in  the  environment 


Future  Directions:  Determining  the  functional  specialization  in  the  nervous  system  will  be 
important  for  understanding  how  the  nervous  system  efficiently  allocates  resources  for  executing 
complex  behaviors.  Although  our  research  is  no  longer  supported  by  the  Office  of  Naval 
Research,  we  have  be  able  to  continue  some  research  on  these  questions  with  the  support  of  the 
McKnight  Foundation  and  the  National  Alliance  for  Research  on  Schizophrenia  and  Depression. 
We  are  currently  continuing  our  studies  of  memory-related  activity  in  cerebral  cortex  with  the  goal 
of  establishing  the  origin  and  distribution  of  these  signals.  Our  efforts  is  focused  on  two  projects. 
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Functional  Specialization  of  the  Temporal  and  Parietal  Pathways  for  Memories.  The 
results  described  above  showed  that  there  were  differences  in  the  contributions  of  the  temporal  and 
parietal  pathways  to  short  term  memories  related  to  performing  a  match  to  sample  task.  We  are 
now  performing  a  direct  comparison  of  temporal  and  parietal  cortex  using  two  tasks  that  we 
believe  are  likely  to  differentiate  the  special  contributions  of  these  regions.  We  have  trained 
animals  to  perform  a  task  in  which  they  must  remember  either  the  location  or  the  shape  of  a  sample 
stimulus.  They  must  later  make  an  either  movement  to  the  remembered  shape  or  location  when  an 
array  of  test  stimuli  appear.  We  are  recording  from  both  temporal  and  parietal  regions.  It  is  our 
expectation  that  we  will  be  able  to  prove  that  neurons  in  parietal  cortex  are  preferentially  involved 
when  the  animals  must  remember  the  location  of  the  target,  and  that  neurons  in  temporal  cortex 
will  be  preferentially  involved  in  remembering  target  shape.  This  project  represents  the  first  direct 
examination  of  region  specialization  for  memory  related  neural  signals. 

Neural  Representations  of  Remembered  Targets.  Given  the  failure  to  demonstrate 
involvement  of  parietal  cortex  in  representing  remembered  colors  or  motion  in  the  match-to-sample 
task  described  above,  we  have  been  investigating  classes  of  memory  to  which  parietal  cortex  might 
contribute.  We  have  recently  discovered  that  a  large  proportion  of  parietal  neurons  are  involved  in 
representing  the  motion  of  objects  that  are  tempor^y  lost  from  view. 

Moving  objects  in  the  environment  frequently  disappear  from  view  by  passing  behind  an 
occluding  objects.  We  nevertheless  can  accurately  extrapolate  the  movements  of  such  objects 
while  they  are  not  in  sight.  We  have  recorded  the  activity  of  neurons  in  the  parietal  cortex  of 
monkeys  while  they  preform  a  task  that  requires  them  to  keep  track  of  the  location  of  targets,  even 
when  those  targets  disappear  for  a  second  or  two.  To  our  surprise,  a  large  proportion  (~40%)  of 
neurons  continued  to  fire  action  potentials  when  the  stimulus  disappeared,  typically  at  a  rate  about 
half  as  great  as  when  the  stimulus  was  present.  These  same  neurons  were  not  active  when  the 
same  stimulus  disappeared  on  control  tri^s  in  which  the  animal  did  not  expect  the  stimulus  to  keep 
moving.  We  believe  that  these  neurons  provide  the  basis  for  an  internal  representation  that  keeps 
track  of  targets  in  the  world,  whether  or  not  they  are  visible. 
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Sclar,  G.,  Maunsell,  Lennie,  P  (1990)  Coding  of  image  contrast  in  central  visual 

pathways  of  the  macaque  monkey.  Vision  Res.  30: 1-10. 

Merigan,  W.H.,  Maunsell,  J.H.R.  (1990)  Macaque  vision  after  magnocellular  lateral 
geniculate  lesions,  ^^sual  Neurosci.  5:347-352. 

Merigan,  W.H.,  Katz,  L,  M.,  Maunsell,  J.H.R.  (1991)  The  effects  of  parvocellular  lateral 
geniculate  lesions  on  the  acuity  and  contrast  sensitivity  of  macaque  monkeys.  J . 
Neurosci.  11:994-1001. 

Maunsell,  J.H.R.,  Sclar,  G.,  Nealey,  T.A.,  DePriest,  D.D.  (1991)  Extraretinal 
representation  by  neurons  in  area  V4  in  the  visual  cortex  of  the  macaque  monkey. 
Visual  Neurosci.  7:561-573. 

Maunsell,  J.H.R.  (1992)  Functional  visual  streams.  Current  Opinion  Neurobiol. 
2:506-510. 

Ferrera,  VP,  Nealey,  T.A.,  Maunsell,  J.H.R.  (1992)  Mixed  parvocellular  and 
magnocellular  geniculate  signals  in  visual  area  V4.  Nature  358:756-758. 

Maunsell,  J.H.R.,  Gibson,  J.R.  (1992)  Visual  response  latencies  in  striate  cortex  of  the 
macaque  monkey.  J.  Neurophysiol.  68:1332-1344. 

Merigan,  W.H.,  Nealey,  T.A.,  Maunsell,  J.H.R.  (1993)  Vsual  effects  of  lesions  of 
cortical  area  V2  in  macaques.  J.  Neurosci.  13:3180-3191. 

Merigan,  W.H.,  Maunsell,  J.H.R.  (1993)  How  parallel  are  the  primate  visual  pathways? 
Ann.  Rev.  Neurosci.  16:369-402. 

Page,  W.K.,  King,  W.M.,  Merigan,  W.,  Maunsell,  J.H.R.  (1994)  Magnocellular  or 
parvocellular  lesions  in  the  lateral  geniculate  nucleus  of  monkeys  cause  minor 
deficits  of  smooth  pursuit  eye  movements.  Msion  Res.  34:223-239. 

Papers  in  Referred  Journals  from  work  during  Funding  Period;  In  press: 

Ferrera,  VP.,  Rudolph,  K.,  Maunsell,  J.H.R.  (1994)  Responses  of  neurons  in  the  parietal 
and  temporal  visual  pathways  during  a  motion  task.  J.  Neurosci.  lAJin  press). 


Book  Chapters  Published  from  work  during  Funding  Period: 

Maunsell,  J.H.R.,  Nealey,  T.A.,  Sclar,  G.,  DePriest,  D.D.  (1989)  Representation  of 
extraretinal  information  in  monkey  visual  cortex  (in)  D.  L&m  (ed.)  Proceedings 
of  the  Retinal  Research  Foundation  Symposium  2:223-235. 

Maunsell,  J.H.R.,  Hochstein,  S.  (1991)  Effects  of  behavioral  state  on  the  stimulus 
selectivity  of  neurons  in  area  V4  of  the  macaque  monkey,  (in)  B.  Blum  (ed.) 
Channels  in  the  Visual  Nervous  System:  Neurophysiology, 
Psychophysics  and  Models.  Freund  Publishing,  Tel  Aviv.  447-470. 

Maunsell,  J.H.R.  (1993)  Neuronal  correlates  of  object  representation,  (in)  T.  Poggio,  D. 
Glaser,  (eds.)  Exploring  Brain  Functions:  Models  in  Neuroscience. 
John  Wiley  and  Sons,  Ltd.,  195-202. 

Maunsell,  J.H.R.,  Ferrera,  VP.  (1993)  Extraretinal  representations  in  visual  areas  of 
macaque  cerebral  cortex,  (in)  T.  Ono,  L.R.  Squire,  M.E.  Raichle,  D.l.  Perrett,  M. 
Fukuda  (eds.)  Brain  Mechanisms  of  Perception  and  Memory:  From 
Neuron  to  Behavior.  Oxford  University  Press,  Oxford,  pp.  104-118. 
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Maunsell,  Ferrera,  VP.  (1994)  Attentional  mechanisms  in  visual  cortex,  (in)  L.R. 

Squire  et  al.  (eds.)  The  Cognitive  Neurosciences,  MIT  Press,  Cambridge, 
MA.  pp.  451-461. 

Maunsell,  J.H.R.,  Ferrera,  VP.  (1994)  Parallel  processing  in  monkey  extrastriate  cortex. 
(in)  T.B.  Lawton  (ed.)  Computation  Vision  Based  on  Neurobiology, 
Proceedings  of  SPIE  2054,  pp.240-242. 

Invited  Presentations  during  the  Funding  Period: 

Optical  Society  of  America  Symposium  on  Functional  Specialization  in  Msual  Cortex, 
Orlando,  October  1989. 

Electrotechnical  Laboratory,  Agency  of  Industrial  Science  and  Technology,  Tsukuba 
Science  City,  Japan,  January  1990. 

Electrotechnical  Laboratory,  Agency  of  Industrial  Science  and  Technology,  Tsukuba 
Science  City,  Japan,  January  1990. 

RIKEN  Institute,  Laboratory  for  Neural  Information  Processing,  Wako,  Japan,  January 
1990. 

Nihon  University  School  of  Medicine,  Department  of  Physiology,  Tokyo,  Japan,  January 
1990. 

National  Institute  for  Physiological  Sciences,  Okazaki,  Japan,  January,  1990. 

International  School  for  Advanced  Studies  Workshop  on  Computational  and  Biological 
Models  of  Visual  Processing,  Trieste,  Italy,  February  1990. 

Office  of  Naval  Research  Workshop  on  Computational  Neuroscience,  Woods  Hole, 
August  1990. 

State  University  of  New  York  at  Buffalo,  Department  of  Biophysical  Sciences,  September 
1990. 

Integrative  and  Behavioral  Neuroscience  Group  Symposium  on  Sensorimotor  Integration, 
Society  for  Neuroscience  Annual  Meeting,  St.  Louis,  October  1990. 

Columbia  University,  Department  of  Biological  Science,  New  Yoik,  April  1991. 

Harvard  Medical  School,  Program  in  Neuroscience  Annual  Retreat,  New  Seabury,  MA, 
April  1991. 

Johns  Hopkins  University,  Department  of  Neuroscience,  June  1991. 

University  of  California  San  Francisco,  Department  of  Physiology  Spring  Symposium, 
June  1991. 

Baylor  College  of  Medicine,  Division  of  Neuroscience,  July  1991. 

International  Brain  Research  Organization  Symposium  on  Parallel  Pathways  in  Vision, 
Montreal,  August  1991. 

International  Symposium  on  Brain  Mechanisms  of  Perception  and  Memory,  Toyama, 
Japan,  October  1991. 

McKnight  Foundation  Conference  on  Neuroscience,  Woods  Hole,  April  1992. 

Association  for  Research  in  Vision  and  Ophthalmology  Symposium  on  Parallel  \^ual 
Channels  in  Primate,  Sarasota,  Florida,  May  1992. 

FASEB  Summer  Research  Conference  on  Biology,  Chemistry  and  Modelling  of  Vision; 
Visual  Processing,  Saxtons  River,  VT,  June  1992. 

Volunteered  Presentations  during  the  Funding  Period: 

Maunsell,  J.H.R.  (1989)  Motion  processing  in  visual  cortex.Opt,  Soc.  Am.  Tech. 
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Invest.  Ophthalmol.  Vis.  Sci.  32:824. 

Pasternak,  T,  Maunsell,  J.H.R.,  Polashenski,  W.,  Merigan,  W.H.,  (1991)  Deficits  in 
global  motion  perception  after  MT/MST  lesions  in  a  macaque.  Invest. 
Ophthalmol.  Ms.  Sci.  32:824. 

Page,  W.K.,  King,  W.M.,  Merigan,  W.H.,  Maunsell,  J.H.R.,  (1991)  Motion  detection 
deficits  revealed  by  step  ramp  tracking  errors  in  monkeys  with  magnocellular  LGN 
lesions.  Invest.  Ophthalmol.  Vis.  Sci.  32:1021. 

Ferrera,  VP,  Maunsell,  J.H.R.  (1991)  Responses  of  single  units  in  macaque  VI  to 
moving  patterns.  Soc.  Neurosci.  Abstr.  17:177. 

Merigan,  W.H.,  Pasternak,  X,  Ferrera,  V,  Maunsell,  J.H.R.  (1991)  Permanent  deficits  in 
speed  discrimination  after  MT/MST  lesions  in  macaque  monkeys.  Soc. 
Neurosci.  Abstr.  17:8. 

Pasternak,  X,  Maunsell,  J.H.R.,  Ferrera,  V,  Merigan,  W.H.  (1991)  Global  motion 
perception  after  MT/MST  lesions  in  a  macaque.  Soc.  Neurosci.  Abstr.  17:8. 

Page,  W.K.,  King,  W.M.,  Merigan,  W.H.,  Maunsell,  J.H.R.  (1991)  Eye  movement 
deficits  revealed  by  step  ramp  tracking  errors  in  monkeys  with  magno  and 
parvocellular  LGN  lesions.  Soc.  Neurosci.  Abstr.  17:860. 

Nealey,  T.A.,  Maunsell,  J.H.R.,  Merigan,  W.H.  (1992)  Visual  function  after  a  lesion  of 
cortical  area  V2  in  the  macaque.  Invest.  Ophthalmol.  Vis.  Sci.  33:1130. 

Training  Activity  during  the  Funding  Period: 

GRADUATE  STUDENTS: 


^Tara  A.  Nealey,  Ph.D.  Physiology,  April  1992,  “Magnocellular  and  parvocellular 
contributions  to  cytochrome  oxidase  compartments  in  VI  of  the  macaque  monkey”. 

Jay  R.  Gibson,  Graduate  Student  in  Neuroscience,  1990  to  present,  “Three-dimensional 
shape  representation  in  area  TE  of  the  behaving  monkey”. 

POSTDOCTORAL  FELLOWS: 

Vincent  P.  Ferrera,  Ph.D.  University  of  Chicago,  1989  to  1992,  “Behavioral  modulation  of 
single  unit  activity  in  area  MT’. 

John  A.  Assad,  Ph.D.  Harvard  University,  1991  to  present,  “The  role  of  posterior  parietal 
cortex  in  target  selection”. 

^Sidney  R.  Lehky,  Ph.D.  University  of  Chicago,  1991  to  1994,  “Neurophysiological 
basis  of  binocular  rivalry”. 

^female,  ^minority 
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